.TH mfsmaster.cfg "5" "November 2018" "MooseFS 3.0.103-1" "This is part of MooseFS"
.SH NAME
mfsmaster.cfg \- main configuration file for \fBmfsmaster\fP
.SH DESCRIPTION
The file \fBmfsmaster.cfg\fP contains configuration of MooseFS master process.
.SH SYNTAX
.PP
Syntax is:
.TP
.IB OPTION " = " VALUE
.PP
Lines starting with \fB#\fP character are ignored as comments.
.SH OPTIONS
Configuration options:
.TP
.B WORKING_USER
user to run daemon as
.TP
.B WORKING_GROUP
group to run daemon as; optional value - if empty then default user group will be used
.TP
.B SYSLOG_IDENT
name of process to place in syslog messages; default is mfsmaster
.TP
.B LOCK_MEMORY
whether to perform mlockall() to avoid swapping out mfsmaster process; default is 0, i.e. no
.TP
.B LIMIT_GLIBC_MALLOC_ARENAS
limit malloc arenas to given value - prevents server from using huge amount of virtual memory (Linux onty, default is 4)
.TP
.B DISABLE_OOM_KILLER
disable out of memory killer (Linux only, default is 1)
.TP
.B NICE_LEVEL
nice level to run daemon with; default is -19; note: process must be started as root to increase priority, if setting of priority fails, process retains the nice level it started with
.TP
.B FILE_UMASK
set default umask for group and others (user has always 0); default is 027 - block write for group and block all for others
.TP
.B DATA_PATH
where to store metadata files and lock file
.TP
.B EXPORTS_FILENAME
alternate location/name of \fBmfsexports.cfg\fP file
.TP
.B TOPOLOGY_FILENAME
alternate location/name of \fBmfstopology.cfg\fP file
.TP
.B LICENCE_FILENAME
alternate location/name of \fBmfslicence.bin\fP file (pro version only)
.TP
.B BACK_LOGS
number of metadata change log files (default is 50)
.TP
.B METADATA_SAVE_FREQ
how often (in hours) master will store metadata (default is 1)
.TP
.B METADATA_DOWNLOAD_FREQ
how often (in hours) leader will download metadata from followers (pro version only ; default is 24)
.TP
.B METADATA_DOWNLOAD_LIMIT
metadata downloading limit in mebibytes per second (pro version only ; 0 means no limit, default is 50 - 50MiB/s = ~52.4MB/s)
.TP
.B BACK_META_KEEP_PREVIOUS
number of previous metadata files to be kept (default is 1)
.TP
.B CHANGELOG_PRESERVE_SECONDS
how many seconds of change logs have to be preserved in memory (default is 1800; 
this sets the minimum, actual number may be a bit bigger due to logs being kept 
in 5k blocks; zero disables extra logs storage)
.TP
.B MISSING_LOG_CAPACITY
how many missing chunks will be stored in master (up to 100*MISSING_LOG_CAPACITY bytes of memory will be allocated ; default value is 100000)
.TP
.B MATOML_LISTEN_HOST
IP address to listen on for metalogger, masters and supervisors connections (* means any)
.TP
.B MATOML_LISTEN_PORT
port to listen on for metalogger, masters and supervisors connections
.TP
.B MASTER_HOST
MooseFS master host, IP is allowed only in single-master installations (pro version only ; default is mfsmaster)
.TP
.B MASTER_RECONNECTION_DELAY
delay in seconds before next try to reconnect to master-leader if not connected (pro version only ; default is 5)
.TP
.B MASTER_TIMEOUT
timeout in seconds for master-leader connections (pro version only ; default is 10)
.TP
.B BIND_HOST
local address to use for connecting with master-leader (pro version only ; default is *, i.e. default local address)
.TP
.B MATOCS_LISTEN_HOST
IP address to listen on for chunkserver connections (\fB*\fP means any)
.TP
.B MATOCS_LISTEN_PORT
port to listen on for chunkserver connections
.TP
.B MATOCS_TIMEOUT
default timeout in seconds for master-chunkserver connection (default is 10)
.TP
.B AUTH_CODE
Optional authentication string. When defined - then only chunkservers with the same AUTH_CODE are allowed to connect to this master. When not defined (default) - then all chunkservers are allowed. If you want to switch on chunkserver authentication, then first define AUTH_CODE in all your chunkservers (and reload/restart them) then define this option and master and reload/restart it. Remember that after reload currently connected chunkservers are NOT disconnected. New AUTH_CODE will be used only when chunkservers will make new connection.
.TP
.BR REMAP_BITS ", " REMAP_SOURCE_IP_CLASS ", " REMAP_DESTINATION_IP_CLASS
Optional IP class remapping. Remap chunkserver IP addresses with first REMAP_BITS of IP equal to first REMAP_BITS of REMAP_SOURCE_IP_CLASS. During remapping system will set first REMAP_BITS of IP address to first REMAP_BITS of REMAP_DESTINATION_IP_CLASS. (defaults are zeros which means no remap)
.TP
.B REPLICATIONS_DELAY_INIT
initial delay in seconds before starting replications (default is 60)
.TP
.B REPLICATIONS_RESPECT_TOPOLOGY
whether to make undergoal replications respect topology (default is 0)
.br
0 - do not respect topology
.br
1 - pick destination server at random, but then choose best source server
.br
2 - try to find destination server in the same rack as one of the existing copies and then replicate chunk locally (in the same rack)
.TP
.B CHUNKS_LOOP_MAX_CPS
Chunks loop shouldn't check more chunks per seconds than given number (default is 100000)
.TP
.B CHUNKS_LOOP_MIN_TIME
Chunks loop shouldn't be done in less seconds than given number (default is 300)
.TP
.B CHUNKS_SOFT_DEL_LIMIT
Soft maximum number of chunks to delete on one chunkserver (default is 10)
.TP
.B CHUNKS_HARD_DEL_LIMIT
Hard maximum number of chunks to delete on one chunkserver (default is 25)
.TP
.B CHUNKS_WRITE_REP_LIMIT
Maximum number of chunks to replicate to one chunkserver (default is 2,1,1,4 - see NOTES)
.TP
.B CHUNKS_READ_REP_LIMIT
Maximum number of chunks to replicate from one chunkserver (default is 10,5,2,5 - see NOTES)
.TP
.B CS_HEAVY_LOAD_THRESHOLD
Threshold for chunkserver load. (default is 150 - see NOTES)
.TP
.B CS_HEAVY_LOAD_RATIO_THRESHOLD
Threshold ratio for chunkserver load (default is 3.0 - see NOTES)
.TP
.B CS_HEAVY_LOAD_GRACE_PERIOD
Defines how long chunkservers will remain in 'grace' mode (default is 900 - see NOTES)
.TP
.B CS_MAINTENANCE_MODE_TIMEOUT
Maximum number of seconds server can be in maintenance mode (default value is 0 - which means 'forever')
.TP
.B CS_TEMP_MAINTENANCE_MODE_TIMEOUT
Maximum number of seconds server can be in "temporary" maintenance mode (server is switched to this mode whenever is stopped gracefully, after reconnection server is switched back to normal mode automatically ; default value: 1800)
.TP
.B ACCEPTABLE_DIFFERENCE
Maximum difference between space usage of chunkservers (deprecated, use \fBACCEPTABLE_PERCENTAGE_DIFFERENCE\fP instead)
.TP
.B ACCEPTABLE_PERCENTAGE_DIFFERENCE
Maximum percentage difference between space usage of chunkservers (default is 1 = 1%)
.TP
.B PRIORITY_QUEUES_LENGTH
Length of priority queues (for endangered, undergoal etc. chunks - chunks that should be processed first - default is 1000000)
.TP
.B MATOCL_LISTEN_HOST
IP address to listen on for client (mount) connections (\fB*\fP means any)
.TP
.B MATOCL_LISTEN_PORT
port to listen on for client (mount) connections
.TP
.B SESSION_SUSTAIN_TIME
How long to sustain a disconnected client session (in seconds; default is 86400 = 1 day)
.TP
.B QUOTA_TIME_LIMIT
Grace period in secods for soft quota (deprecated, use \fBQUOTA_DEFAULT_GRACE_PERIOD\fP instead for default value or specify it individually)
.TP
.B QUOTA_DEFAULT_GRACE_PERIOD
Default grace period in seconds for soft quota (default is 604800 = 7 days)
.TP
.B ATIME_MODE
Set atime modification mode (default is 0 = always modify atime - see NOTES)
.TP
.B RESERVE_SPACE
Set amount of space reserved for superuser (default is 0 = do not reserve space for superuser - see NOTES)
.SH NOTES
.PP
Chunks in master are tested in a loop. Speed (or frequency) is regulated by two
options \fBCHUNKS_LOOP_MIN_TIME\fP and \fBCHUNKS_LOOP_MAX_CPS\fP. First
defines minimal time between iterations of the loop and second defines 
maximal number of chunk tests per second. 
Typically at the beginning, when number of chunks is small, time is
constant, regulated by \fBCHUNK_LOOP_MIN_TIME\fP, but when number of chunks
beccomes bigger then time of loop can increase according to
\fBCHUNKS_LOOP_MAX_CPS\fP.
.PP
Example: \fBCHUNKS_LOOP_MIN_TIME\fP is set to 300, \fBCHUNKS_LOOP_MAX_CPS\fP 
is set to 100000 and there is 1000000 (one million) chunks in the system. 1000000/100000 = 10, 
which is less than 300, so one loop iteration will take 300 seconds.
With 1000000000 (one billion) chunks the system needs 10000 seconds for one iteration of the loop.
.PP
Deletion limits are defined as 'soft' and 'hard' limit. When number of chunks
to delete increases from loop to loop, current limit can be temporary
increased above soft limit, but never above hard limit.
.PP
Replication limits are divided into four cases:
.IP \[bu] 2
first limit is for endangered chunks (chunks with only one copy)
.IP \[bu] 2
second limit is for undergoal chunks (chunks with number of copies lower than specified goal)
.IP \[bu] 2
third limit is for rebalance between servers with space usage around arithmetic mean
.IP \[bu] 2
fourth limit is for rebalance between other servers (very low or very high space usage)
.PP
Usually first number should be grater than or equal to second, second greater than or equal to third, and fourth greater than or equal to third ( 1st >= 2nd >= 3rd <= 4th ). If one number is given, then all limits are set to this number (for backward compatibility).
.PP
Whenever chunkserver load is higher than \fBCS_HEAVY_LOAD_THRESHOLD\fP  and \fBCS_HEAVY_LOAD_RATIO_THRESHOLD\fP times higher than average load, then chunkserver is switched into 'grace' mode. Chunkserver stays in grace mode for \fBCS_HEAVY_LOAD_GRACE_PERIOD\fP seconds.
.PP
There are five values for \fBATIME_MODE\fP (all other values are treated as 0):
.IP \[bu] 2
\fB0\fP = Always modify atime for files, folders and symlinks.
.IP \[bu] 2
\fB1\fP = Always modify atime but only in case of files (do not modify atime in case of folders and symlinks).
.IP \[bu] 2
\fB2\fP = Modify atime only when it is lower than ctime or mtime and when current time is higher than ctime or mtime respectively, also modify atime when current atime is older than 24h. Do it for all objects during access (like "relatime" option in Linux).
.IP \[bu] 2
\fB3\fP = Same as above but only in case of files. In case of folders and symlinks do not modify atime.
.IP \[bu] 2
\fB4\fP = Never modify atime during access (like "noatime" option).
.PP
You can reserve space for superuser using \fBRESERVE_SPACE\fP option. You can define it as number of bytes, percent of total space, capacity of biggest chunkserver, etc.
.IP \[bu] 2
\fI#\fP or \fI#\fP\fBB\fP = number of bytes reserved for superuser. Standard metric prefixes can be used - SI and IEC (k,K,M,Mi,G,Gi etc.)
.IP \[bu] 2
\fI#\fP\fB%\fP or \fI#\fP\fB.\fP\fI#\fP\fB%\fP = percent of total capacity of MooseFS instance
.IP \[bu] 2
\fI#\fP\fBU\fP or \fI#\fP\fB.\fP\fI#\fP\fBU\fP = multiplies of "U" value; U is defined as maximum number of bytes currently used by a single chunkserver
.IP \[bu] 2
\fI#\fP\fBC\fP or \fI#\fP\fB.\fP\fI#\fP\fBC\fP = multiplies of "C" value; C is defined as maximum total capacity of a single chunkserver
.PP
When your network has two (or more) IP classes you may want to use one network for standard communication between MFS modules and separate network only for I/O. It can be done by setting REMAP_BITS, REMAP_SOURCE_IP_CLASS and REMAP_DESTINATION_IP_CLASS.
When you set these options then master will change internally IP addresses of chunkservers and will send them as chunk locations, so clients will make connections with chunkservers using new (destination) IP for all I/O, but still communicate with master using original (source) IP.
Also chunkservers will use original IP to communicate with master, but they will use new IP's to communicate between themselves during replication. Beware that all clients and chunkservers must have access to both networks, but masters, metaloggers etc. will need only access to the source network.
.SH COPYRIGHT
Copyright (C) 2018 Jakub Kruszona-Zawadzki, Core Technology Sp. z o.o.

This file is part of MooseFS.

MooseFS is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 2 (only).

MooseFS is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with MooseFS; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301, USA
or visit http://www.gnu.org/licenses/gpl-2.0.html
.SH "SEE ALSO"
.BR mfsmaster (8),
.BR mfsexports.cfg (5)
.BR mfstopology.cfg (5)
